Sandro.Quaranta.nl

Software delivery, how do the BIG guys do it?

Successful agile software delivery (nowadays buzzed as DEVOPS) starts with a complete agile Portfolio & Release organisation where both the corporate culture and the ICT architecture have to be in service of agility.
On the software development level the key success factors for continuous delivery are the solutions architecture and automated testing which have to be completely designed towards the agile and automated delivery practices.
All todays BIG successful companies fully comply with this recipe.

Microsoft BING

The agile approach of combining development and testing, under the name "combined engineering" (first used in the Bing team), is also spreading. At Bing, the task of creating programmatic tests was moved onto developers, instead of dedicated testers. QA still exists and is still important, but it performs end-user style "real world" testing, not programmatic automated testing. This testing has been successful for Bing, improving the team's ability to ship changes without harming overall software quality. Ratio developers / testers went from 1:2 to 1:1

Methodology

Continuous delivery by feature teams with 600 developers
1 code branche,
Automated testing is key. Developers can only check-in when all tests passed successfully.

Environments|

Locally developed features committed to the main branche after having passed full automated test cycle Build service in the cloud
Production where also the tests are run

Roles

Software Design Engineer

Write good and sustainable code, design modularly and understand technical implications of such design choices.

Software Design Engineer in Test

A Software Design Engineer in Test (SDET) at Microsoft is a developer with the primary responsibility of writing code and tools to test products.

Program Manager

Partner with development and work through the entire product cycle as the advocate for end-users and customers

UX Designer

Informing and envisioning the product plans, to driving success metrics and detailed designs,

Product Planner

Product Planning is a shared discipline within both the Marketing and Engineering professions

Release frequency

deployment to PRD 20 x week,

Sources
BING continuous delivery
MS application services
Linkedin

Google

Google is applying the “dogfooding”practices and Beta release approach for their Office tools. Canary releases where all applications are first used by a limited audience of (internal) leading users, before spreading them over the world. Before releasing internally, extensive testing has been done by the Google test teams. Source Dogfooding at Google Testing at GOOGLE Google release pipeline

Methodology

Dogfooding Continuous delivery almost everything in Google is developed on mainline.

Environments

For smaller web apps in cloud: Laptop > STG > PRD
For Office tools: DEV > TST > Canary > Beta > PRD

Release frequency

Many Google services see releases multiple times a week,

Facebook

Release frequency

Facebook releases to production twice a day.

Amazon

Environments

development, staging and production environments for Amazon’s digital music retail website DEV > STG > PRD

Release frequency

Amazon is on record as making changes to production every 11.6 seconds on average in May of 2011. Sources Software Engineer in Test (Systems Support) vacancy

Spotify

our workflow from local development, via staging and canary, to production. Local DEV > STG > Canary > PRD Manual deployments, DEV accompanies releases to OPS in person, “get it LIVE aqap” don’t waist time in full-automation. containers are already a way of life. The streaming-music provider has been using containers in production on a large scale. With Docker they can build an image, test that image and then use that same image in production. Cat fooding: use competitors products Sources Software Engineering - Career opportunities – Spotify vacancy

Unbounce

QA and DEVOPS at Unbounce

But as the lines blur between product and operation – as the very name DevOps implies – it’s no great leap to recognize that the environment itself is a part of the product.

Carl continues, “It’s QA’s responsibility to actually push new code out to production, so the DevOps team has been providing them with tools that make blue-green deployments push-button easy. Our QA team can then initiate deploys, verify that the intended change functions as expected, cut over to the newly deployed code, and also roll back if there is any reported issue.“ DevOps QA Is About Preventing Defects, Not Finding Them

QA takes a critical role in this organizational structure because they have the visibility and the directive to push code out when it is working, and roll it back when it is not. This is a very different mindset from QA organizations of 10 years ago, whose primary responsibilities involved finding bugs. Today QA groups are charged with preventing defects from reaching the public site. This has several implications:
•QA owns continuous improvement and quality tracking across the entire development cycle. They are the ones who are primarily responsible for identifying problems not just in the product but also in the process, and recommending changes wherever they can.
•Tests are code, as any test automation expert will tell you. It’s a necessity, of course. If your process is designed to publish a new release every day (or every hour) there is no room for manual testing. You must develop automation systems, through code, that can ensure quality standards are maintained.
•Automation rules. Anything that can be automated, should be automated. When Carl describes Unbounce’s deployment process as “push-button easy,” this is what he’s talking about.
•Testers are the quality advocates, influencing both development and operational processes. They don’t just find bugs. They look for any opportunity to improve repeatability and predictability.
Beyond Functional Testing: Automation for Load Testing, Stress Testing, and Performance Testing

Now we are at a very exciting time in the transformation of QA, because while many organizations have mature processes for automating functional testing, they are only just beginning to apply these practices to other areas of testing like security and stress testing.

In particular, load testing and stress testing are critical disciplines for DevOps organizations that are moving at high velocity. A bottleneck introduced to a critical transaction process on an eCommerce website can bring a business to its knees. You want to do everything you can to identify scaling problems before a product is pushed out to a production environment, and you also want to keep a close eye on performance after it’s been released.

If you are curious to understand how your process holds up, check out this infographic: How Automated Your Performance Testing Is. Where does QA fit in DevOps QA for DevOps is, and probably always will be, a broad term. If you can't independently and automatically validate your infrastructure and application configuration though then there's a massive gap in your approach.